ARTICLE  CRITIQUE  CHECKLIST 


INTRODUCTION 

Problem  Yes  No 

Is  there  a statement  of  the  problem  or  a qualitative 

topic  of  study?  

Does  the  problem  of  topic  indicate  a particular  focus  of  study?  

Is  the  problem  “researchable,”  that  is,  can  it  be  investigated 

through  the  collection  and  analysis  of  data?  

Is  the  background  information  on  the  problem  presented?  

Is  the  educational  significance  of  the  problem  discussed?  

Does  the  quantitative  problem  statement  indicate  the 
variables  of  interest  and  the  specific  relationship  between 

those  variables  that  were  investigated?  

Does  the  qualitative  problem  statement  provide  a general 

indication  of  the  research  topic  or  issue?  

When  necessary,  are  variables  directly  or  operationally 

defined?  


Does  the  researcher  have  the  knowledge  and  skill  to  carry 
out  the  proposed  research? 

Review  of  the  Related  Literature 

Is  the  review  comprehensive? 

Are  all  references  cited  relevant  to  the  problem  under 
investigation? 

Are  most  of  the  sources  primary;  i.e.,  are  there  only  a few 
or  no  secondary  sources? 

Have  the  references  been  analyzed  and  critiqued,  and  the 
results  of  various  studies  compared  and  contrasted?  That 
is,  is  the  review  more  than  a series  of  abstracts  or  annotations? 


Is  the  relevancy  of  each  reference  explained? 

Is  the  review  well  organized? 

Does  it  logically  flow  in  such  a way  that  the  references  least 
related  to  the  problem  are  discussed  first  and  the  most 
related  references  are  discussed  last? 


N/A 


Yes 


No 


N/A 


Does  it  educate  the  reader  about  the  problem  or  topic? 

Does  the  review  conclude  with  a summary  and  interpretation 
of  the  literature  and  its  implications  for  the  problem 
investigated? 

Do  the  implications  discussed  form  an  empirical  or  theoretical 
rationale  for  the  hypotheses  that  follow? 

Are  references  cited  completely  and  accurately? 

Hypothesis 

Are  specific  questions  to  be  answered  listed  or  specific 
hypotheses  to  be  tested  stated? 

Does  each  hypothesis  state  an  expected  relationship  or 
difference? 

If  necessary,  are  variables  directly  or  operationally  defined? 

Is  each  hypothesis  testable? 

METHOD 

Participants 

Are  the  size  and  major  characteristics  of  the  population 
studied  described? 

Are  the  accessible  and  target  populations  described? 

If  a sample  was  selected,  is  the  method  of  selecting  the 
sample  clearly  described? 

Does  the  method  of  sample  selection  suggest  any  limitations 
or  biases  in  the  sample? 

For  example,  is  stratified  sampling 
used  to  obtain  sample  subgroups? 

Are  the  size  and  major  characteristics  of  the  sample  described? 

Does  the  sample  size  meet  the  suggested  guideline  for  minimum 
sample  size  appropriate  for  quantitative  analyses? 

Instruments 

Do  instruments  and  their  administration  meet  guidelines  for 
protecting  human  subjects? 


Yes 


No 


N/A 


Have  needed  permissions  been  obtained? 

Is  the  rationale  given  for  the  selection  of  the  instruments  (or 
measurements)  used? 

Is  each  instrument  described  in  terms  of  purpose,  content, 
validity,  and  reliability? 

Are  the  instruments  appropriate  for  measuring  the  intended 
variables? 

Does  the  researcher  have  the  needed  skills  or  experience  to 
construct  or  administer  an  instrument? 

Is  evidence  presented  to  indicate  that  the  instruments  are 
appropriate  for  the  intended  sample?  (For  example,  is  the 
reading  level  of  an  instrument  suitable  for  sample  participants?) 

If  appropriate,  are  subtests  reliabilities  given? 

If  an  instrument  was  developed  specifically  for  the  study, 
are  the  procedures  involved  in  its  development  and  validation 
described? 

If  an  instrument  was  developed  specifically  for  the  study,  are 
administration,  scoring  or  tabulating,  and  interpretation 
procedures  fully  described? 

Is  the  correct  type  of  instrument  used  for  data  collection? 

(For  example,  using  a norm-referenced  instrument  when  a 
criterion-referenced  one  is  more  suitable.) 

Design  and  Procedures 

Are  the  design  and  procedures  appropriate  for  examining  the 
research  question  or  testing  the  hypotheses  of  the  study? 

Are  the  procedures  described  in  sufficient  detail  to  permit 
them  to  be  replicated  by  another  researcher? 

Do  procedures  logically  relate  to  each  other? 

Are  instruments  and  procedures  applied  correctly? 

If  a pilot  study  was  conducted,  are  its  execution  and  results 
described  as  well  as  its  impact  on  the  subsequent  study? 


Are  control  procedures  described? 


Yes  No  N/A 


Did  the  researcher  discuss  or  account  for  any  potentially 
confounding  variables  that  he  or  she  was  unable  to  control? 

Is  the  application  of  the  qualitative  method  chosen  described 
in  detail? 


Is  the  context  of  the  qualitative  study  described  in  detail? 

RESULTS 

Are  appropriate  descriptive  statistics  presented? 

Was  the  probability  level  at  which  the  results  of  the  tests  of 
significance  were  evaluated  specified  in  advance  of  the  data 
analysis? 

Was  every  hypothesis  tested? 

If  parametric  tests  were  used,  is  there  evidence  that  the 
researcher  avoided  violating  the  required  assumptions  for 
parametric  tests? 

Are  the  tests  of  significance  described  appropriate,  given  the 
hypotheses  and  design  of  the  study? 

Was  the  inductive  logic  used  to  produce  results  in  a qualitative 
study  made  explicit? 

Are  the  tests  of  significance  interpreted  using  the  appropriate 
degrees  of  freedom? 

Are  the  results  clearly  described? 

Are  the  tables  and  figures  (if  any)  well  organized  and  easy  to 
understand? 


Are  the  data  in  each  table  and  figure  described  in  the  text?  

DISCUSSION,  CONCLUSIONS,  OR  RECOMMENDATIONS 

Is  each  result  discussed  in  terms  of  the  original  hypothesis  or 

topic  to  which  it  relates?  

Is  each  result  discussed  in  terms  of  its  agreement  or 

disagreement  with  previous  results  obtained  by  other 

researchers  in  other  studies?  


Are  generalizations  consistent  with  the  results? 

Are  the  possible  effects  of  uncontrolled  variables  on  the 
results  discussed? 


Yes  No  N/A 

Are  theoretical  and  practical  implications  of  the  findings 

discussed?  

Are  recommendations  for  future  action  made?  

Are  the  suggestions  for  future  action  based  on  practical 
significance  or  on  statistical  significance  only;  i.e.,  has  the 

author  avoided  confusing  practical  and  statistical  significance?  

ABSTRACT  OR  SUMMARY 

Is  the  problem  restated?  

Are  the  number  and  type  of  subjects  and  instruments 

described?  

Is  the  design  used  identified?  

Are  procedures  described?  

Are  the  major  results  and  conclusions  restated?  


TYPE-SPECIFIC  EVALUATION  CRITERIA 


In  addition  to  general  criteria  that  can  be  applied  to  almost  any 
study,  additional  questions  should  be  asked  depending  on  the 
type  of  research  represented  by  the  study.  In  other  words,  there 
are  concerns  that  are  specific  to  historical  studies,  and  likewise 
to  other  qualitative,  survey,  correlational,  causal-comparative, 
and  experimental  studies. 


Qualitative  Research  in  General  Yes  No 

Does  the  topic  statement  provide  a general  sense  of  the  study 

focus?  


Is  the  purposive  sampling  procedure  described  and  related  to 
the  study  focus? 

Is  each  data  collection  strategy  described? 

Is  the  role  the  researcher  assumed  stated  (e.g.,  observer, 
participant  observer,  interviewer,  etc.)? 

Is  the  research  site  and  the  researcher’s  entry  into  it 
described? 


Were  the  data  collection  strategies  used  appropriately,  given 
the  purpose  of  the  study? 

Were  strategies  used  to  strengthen  the  validity  and  reliability 
of  the  data  (e.g.,  triangulation)? 

Is  there  a description  of  how  any  unexpected  ethical  issues 
were  handled? 


Were  strategies  used  to  minimize  observer  bias  and  observer 
effect  described? 


Are  the  researcher’s  reactions  and  notes  differentiated  from 
descriptive  field  notes? 

Are  data  coding  strategies  described  and  examples  of  coded 
data  given? 

Is  the  inductive  logic  applied  to  the  data  to  produce  results 
stated  in  detail? 


N/A 


Are  conclusions  supported  by  data  (e.g.,  are  direct  quotes 
used  to  illustrate  points  made)? 


Observation  Studies 


Yes  No  N/A 


Are  observational  variables  defined?  

How  were  observers  trained?  

Did  different  observers  work  and  score  independently?  

Were  observers  required  to  observe  only  one  behavior  at  a 

time?  

Was  a coded  recording  instrument  used?  

Are  the  qualifications  and  special  training  of  the  observers 

described?  

Was  the  level  of  interobserver  reliability  sufficiently  high?  

Were  efforts  made  to  overcome  observer  bias  and  observer 

effect?  

Was  observation  of  subjects  the  most  appropriate  approach 

for  data  collection  (as  opposed  to  use  of  some  unobtrusive 

measure)?  

Was  a description  of  how  the  observational  data  were 

analyzed  provided?  

Historical  Research  Yes  No 

Were  the  sources  of  data  related  to  the  problem  mostly 

primary?  

Was  each  piece  of  data  subjected  to  external  criticism?  

Was  each  piece  of  data  subjected  to  internal  criticism?  

Does  the  researcher  examine  the  possibility  of  personal  bias  in 

the  study  analysis  and  conclusion?  

Are  causal  inferences  or  conclusions  warranted  given  the  data 

studied?  

Is  the  report  of  the  study  an  integrated,  synthesized, 

chronological  presentation  of  the  results?  

Survey  — Questionnaire  Studies  Yes  No 

Are  questionnaire  validation  procedures  described?  

Was  the  questionnaire  pretested?  

Are  pilot  study  procedures  and  results  described?  


N/A 


N/A 


Are  directions  to  questionnaire  respondents  clear?  

Does  each  item  in  the  questionnaire  relate  to  one  of  the 

objectives  of  the  study?  

Does  each  questionnaire  item  deal  with  a single  concept?  

When  necessary,  is  a point  of  reference  given  for  questionnaire 

items?  

Does  the  cover  letter  explain  the  purpose  and  importance  of  the 
study  and  give  the  potential  responder  a good  reason  for 

cooperating?  

If  appropriate,  is  confidentiality  or  anonymity  of  responses 

assured  in  the  cover  letter?  

What  is  the  percentage  of  returns  and  how  does  it  affect  the 

study  results?  

Are  followup  activities  to  increase  returns  described?  

If  the  response  rate  was  low,  was  any  attempt  made  to  determine 

any  major  differences  between  responders  and  nonresponders?  

Are  data  analyzed  in  groups  or  clusters  rather  than  a series 

of  many  single  variable  analyses?  

Survey  — Interview  Studies  Yes  No  N/A 

Were  the  interview  procedures  pretested?  

Are  pilot  study  procedures  and  results  described?  

Does  each  item  in  the  interview  guide  relate  to  a specific 

objective  of  the  study?  

When  necessary,  is  a point  of  reference  given  in  the  guide  for 

interview  items?  

Are  leading  questions  avoided  in  the  interview  guide?  

Is  the  language  and  complexity  of  the  questions  appropriate 

for  the  participants?  

Does  the  interview  guide  indicate  the  type  and  amount  of 

prompting  and  probing  that  was  permitted?  

Are  the  qualifications  and  special  training  of  the  interviewers 

described?  

Is  the  method  used  to  record  responses  described?  

Did  the  researcher  use  the  most  reliable,  unbiased  method  of 

recording  responses  that  could  have  been  used?  


Did  the  researcher  specify  how  the  responses  to  semistructured 

and  unstructured  items  were  quantified  and  analyzed?  

Correlation  — Relationship  Studies  Yes  No  N/A 

Were  variables  carefully  selected;  i.e.,  was  a shotgun  approach 

avoided?  

Is  the  rationale  for  variable  selection  described?  

Are  conclusions  and  recommendations  based  on  values  of 
correlation  coefficients  corrected  for  attenuation  or  restriction 

in  range?  

Do  the  conclusions  avoid  suggesting  causal  relationships 

between  the  variables  investigated?  

Correlation — Prediction  Studies  Yes  No  N/A 

Is  a rationale  given  for  selection  of  predictor  variables?  

Is  the  criterion  variable  well  defined?  

Was  the  resulting  prediction  equation  validated  with  at  least 

one  other  group?  

Causal  — Comparative  Studies  Yes  No  N/A 

Are  the  characteristics  or  experiences  that  differentiate  the 

groups  (the  independent  variable)  clearly  defined  or  described?  

Are  critical  extraneous  variables  identified?  

Were  any  control  procedures  applied  to  equate  the  groups  on 

extraneous  variables?  

Are  causal  relationships  found  discussed  with  due  caution?  

Are  plausible  alternative  hypotheses  discussed?  

Experimental  Research  Yes  No  N/A 

Was  an  appropriate  experimental  design  selected?  

Is  a rationale  for  design  selection  given?  

Are  sources  of  invalidity  associated  with  the  design  identified 

and  discussed?  

Is  the  method  of  group  formation  described?  

Was  the  experimental  group  formed  in  the  same  way  as  the 

control  group?  


Were  groups  randomly  formed  and  the  use  of  existing  groups 
avoided? 

Were  treatments  randomly  assigned  to  groups? 

Were  critical  extraneous  variables  identified? 

Were  any  control  procedures  applied  to  equate  groups  on 
extraneous  variables? 

Were  possible  reactive  arrangements  (e.g.,  the  Hawthorne 
effect)  controlled  for? 

Were  tables  clear  and  pertinent  to  the  research  results? 

Were  the  results  generalized  to  the  appropriate  group? 


